Senising on uavs for mapping and obstacle avoidance

ABSTRACT

Structured light approaches utilize a laser to project features, which are then captured with a camera. By knowing the disparity between the laser emitter and the camera, the system can triangulate to find the range. Four, 185 degree field-of-view cameras provide overlapping views over nearly the whole unit sphere. The cameras are separated from each other to provide parallax. A near-infrared laser projection unit sends light out into the environment, which is reflected and viewed by the cameras. The laser projection system will create vertical lines, while the cameras will be displaced from each other horizontally. This relative shift of the lines, as viewed by different cameras, enables the lines to be triangulated in 3D space. At each point in time, a vertical stripe of the world will be triangulated. Over time, the laser line will be rotated over all yaw angles to provide full a 360 degree range.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Patent Application Ser.62/175,231, entitled “SENISING ON UAVS FOR MAPPING AND OBSTACLEAVOIDANCE”, filed on 13 Jun. 2015. The benefit under 35 USC §119(e) ofthe United States provisional application is hereby claimed, and theaforementioned application is hereby incorporated herein by reference.

FEDERALLY SPONSORED RESEARCH

Not Applicable

SEQUENCE LISTING OR PROGRAM

Not Applicable

TECHNICAL FIELD OF THE INVENTION

The present invention relates to UAVs. More specifically, the presentinvention is related to providing structured light and time of flightsensors on UAVs for obstacle avoidance and creating mappingcapabilities.

BACKGROUND OF THE INVENTION

There are few sensors that are well suited for autonomous mobility andmapping functions on small aerial platforms. LADAR choices that can fitthe SWAP requirements are severely limited; few LADARs are availablewithin the SWAP. One option, the single line sensor, needs to beconfigured into an up-down tilt configuration, the so called “yes-yes”ladar, or into a side to side pan configuration, so called “no-no”ladar, in order to get the coverage needed to traverse a complexenvironment.

Some other sensors provide a relatively small vertical field-of-view.Quadrotors of a small size and weight create significant pitch whentraveling at high speeds. This pitch can be as high as 45 degrees whentraveling at high speeds, or when quadrotors are used in windy areas.

Therefore, if a sensor with relatively small vertical field of view isinstalled horizontally, the vehicle will be blind in the direction oftravel at high speeds. Once again, there is a need of a tilt mechanism.

The other approach, which better fits the SWAP constraints of aquadrotor, is stereo vision—or structure from motion. However, in bothcases, poor lighting of an indoor environment—together with the lowerquality optics camera combinations that can be carried with thequads—makes it a poor choice. Many attempts like this have beenperformed in the past few years, with very poor results.

Definitions

LADAR (also known as LIDAR) is an optical remote sensing technology thatcan measure the distance to, or other properties of a target byilluminating the target with light, often using pulses from a laser.LIDAR technology has application in geomatics, archaeology, geography,geology, geomorphology, seismology, forestry, remote sensing andatmospheric physics, as well as in airborne laser swath mapping (ALSM),laser altimetry and LIDAR contour mapping. The acronym LADAR (LaserDetection and Ranging) is often used in military contexts. The term“laser radar” is sometimes used, even though LIDAR does not employmicrowaves or radio waves and therefore is not radar in the strict senseof the word.

In computing, a graphical user interface (GUI, commonly pronouncedgooey) is a type of user interface that allows users to interact withelectronic devices using images rather than text commands. GUIs can beused in computers, hand-held devices such as MP3 players, portable mediaplayers or gaming devices, household appliances and office equipment. AGUI represents the information and actions available to a user throughgraphical icons and visual indicators such as secondary notation, asopposed to text-based interfaces, typed command labels or textnavigation. The actions are usually performed through directmanipulation of the graphical elements.

MAPHAC is a 3D scanning device for measuring the three-dimensional shapeof an object using projected light patterns and a camera system.

A quadcopter, also called a quadrotor helicopter or quadrotor, is amultirotor helicopter that is lifted and propelled by four rotors.Quadcopters are classified as rotorcraft, as opposed to fixed-wingaircraft, because their lift is generated by a set of rotors (verticallyoriented propellers). Unlike most helicopters, quadcopters use two setsof identical fixed pitched propellers; two clockwise (CW) and twocounter-clockwise (CCW). These use variation of RPM to control lift andtorque. Control of vehicle motion is achieved by altering the rotationrate of one or more rotor discs, thereby changing its torque load andthrust/lift characteristics.

A Small Unmanned Ground Vehicle (SUGV) is a lightweight, man portableUnmanned Ground Vehicle (UGV) capable of conducting military operationsin urban terrain, tunnels, sewers, and caves. The SUGV aids in theperformance of manpower-intensive or high-risk functions (i.e. urbanIntelligence, Surveillance, and Reconnaissance (ISR) missions,chemical/Toxic Industrial Chemicals (TIC), Toxic Industrial Materials(TIM), reconnaissance, etc.). Working to minimize Soldiers' exposuredirectly to hazards, the SUGV's modular design allows multiple payloadsto be integrated in a plug and play fashion.

An Unmanned Ground Vehicle (UGV) is a vehicle that operates while incontact with the ground and without an onboard human presence. UGVs canbe used for many applications where it may be inconvenient, dangerous,or impossible to have a human operator present. Generally, the vehiclewill have a set of sensors to observe the environment, and will eitherautonomously make decisions about its behavior or pass the informationto a human operator at a different location who will control the vehiclethrough teleoperation. The UGV is the land-based counterpart to unmannedaerial vehicles and remotely operated underwater vehicles. Unmannedrobotics are being actively developed for both civilian and military useto perform a variety of dull, dirty, and dangerous activities.

SWAP constraints are directed to size, weight, and power of a militaryplatform as defined by the military for a given platform and providing abasis for which a platform and utilize components from variousmanufacturers.

SUMMARY OF THE INVENTION

Structured light approaches utilize a laser to project features, whichare then captured with a camera. By knowing the disparity between thelaser emitter and the camera, the system can triangulate to find therange. In order to accommodate these sensors on a quadrotor,modifications will be done to the location of the camera and the laseremitters as taught by the present invention.

The proposed configuration makes use of multiple fisheye cameras andlaser line scanners. Four, wide degree field-of-view cameras provideoverlapping views over nearly the whole unit sphere. The cameras areseparated from each other to provide parallax. A near-infrared laserprojection unit sends light out into the environment. If the light hitsobjects in the environment it is reflected and viewed by the cameras.

The laser projection system will create vertical lines, while thecameras will be displaced from each other horizontally. This relativeshift (stereo disparity) of the lines, as viewed by different cameras,enables the lines to be triangulated in 3D space. At each point in time,a vertical stripe of the world will be triangulated. Over time, thelaser line will be rotated over all yaw angles to provide full 360degree range sensing capabilities.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein a form a partof the specification, illustrate the present invention and, togetherwith the description, further serve to explain the principles of theinvention and to enable a person skilled in the pertinent art to makeand use the invention.

FIG. 1. MAPHAC is a structured light sensor that is designed for SUGVs.

FIG. 2 Point cloud generated by MAPHAC, color-coded for range. The pointcloud shows a leaning ladder, and a variety of office clutter.

FIG. 3a . Quad copter with four imagers and laser projection system.

FIG. 3b . Approximate field-of-view of a single imager.

FIG. 3c . Overhead view of combined field-of-view of all imagers.

FIG. 3d . Side-view of combined field-of-view of all imagers.

FIG. 4. Two laser line projectors are used to create a line that canthen be sensed with the omnidirectional cameras.

FIG. 5. Complete field of view showing laser and cameras.

FIG. 6. Expected range error of structured light sensor.

FIGS. 7a and 7b . Prototype sensing plane configuration.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the invention of exemplaryembodiments of the invention, reference is made to the accompanyingdrawings (where like numbers represent like elements), which form a parthereof, and in which is shown by way of illustration specific exemplaryembodiments in which the invention may be practiced. These embodimentsare described in sufficient detail to enable those skilled in the art topractice the invention, but other embodiments may be utilized andlogical, mechanical, electrical, and other changes may be made withoutdeparting from the scope of the present invention. The followingdetailed description is, therefore, not to be taken in a limiting sense,and the scope of the present invention is defined only by the appendedclaims.

In the following description, numerous specific details are set forth toprovide a thorough understanding of the invention. However, it isunderstood that the invention may be practiced without these specificdetails. In other instances, well-known structures and techniques knownto one of ordinary skill in the art have not been shown in detail inorder not to obscure the invention. Referring to the figures, it ispossible to see the various major elements constituting the apparatus ofthe present invention.

Structured light approaches utilize a laser to project features, whichare then captured with a camera. By knowing the disparity between thelaser emitter and the camera, the system can triangulate to find therange. In sharp contrast, with conventional stereo and structure frommotion, poor lighting actually improves the range and accuracy of thissensor. There is also no need to have rich features in the environment,since the laser “projects its own features.” Therefore, it will evenwork on featureless walls and floors.

One such approach is presented in FIG. 1, which is currently installedon a SUGV (small unmanned ground vehicle). It is designed to create veryhigh density point clouds for mapping applications at two megapixels persecond. FIG. 1 illustrates where a MAPHAC 100 is a structured lightsensor that is designed for SUGVs.

FIG. 2, shows the scan of a typical cluttered room as a point cloud 200,including a ladder 201, a camera with a tripod 202, chairs 203, lamps204, etc. In FIG. 2 the point cloud 200 generated by an MAPHAC iscolor-coded for range. The point cloud 200 shows a leaning ladder 201,and a variety of office clutter. The current incarnation of MAPHAC 100is designed to become a substitute for a SUGV antenna, where it canserve as both an autonomous mobility sensor and radio antenna.

In order to accommodate these sensors on a quadrotor, modifications willbe done to the location of the camera and the laser emitters. However,the core electronics and software have already been designed, but neverused in this combination. The sensor is designed to meet the uniqueneeds of an autonomous multicopter for indoor and outdoor environments,including: Large-field of view for obstacle avoidance and mapping;Light-weight system with minimal moving parts; Accurate ranges at shortdistances, with decreasing accuracy at longer ranges; Use of eye-safelasers, while providing resilience to ambient light; and a Predictedweight under 150 grams.

The proposed configuration makes use of multiple fisheye cameras andlaser line scanners. Four, 185 degree field-of-view cameras provideoverlapping views over nearly the whole unit sphere. The cameras areseparated from each other to provide parallax. A near-infrared laserprojection unit sends light out into the environment, which is reflectedand viewed by the cameras. The laser projection system will createvertical lines, while the cameras will be displaced from each otherhorizontally. This relative shift (stereo disparity) of the lines, asviewed by different cameras, enables the lines to be triangulated in 3Dspace.

At each point in time, a vertical stripe of the world will betriangulated. Over time, the laser line will be rotated over all yawangles to provide full 360 degree range sensing capabilities asillustrated by FIGS. 3a, 3b, 3c , and 3 d.

FIG. 3a illustrates a Quad copter 300 with four imagers 301, 302, 303,and 304 and laser projection system 305. FIG. 3b illustrates anapproximate field-of-view 306 of a single imager 301. FIG. 3cillustrates an overhead view of combined field-of-view 307 of allimagers 301, 302, 303, and 304. FIG. 3d illustrates a side-view ofcombined field-of-view 308 of all imagers.

FIG. 4 illustrates where two laser line projectors 401 and 402 are usedto create a line 403 that can then be sensed with the omnidirectionalcameras.

Each imager is composed of a camera module, a spectral filter, and awide-angle compound lens. The camera must be small in size and weight,while providing high sensitivity and a wide dynamic range. Depending onmission requirements, an optical bandpass filter can be installed toattenuate incoming ambient light. If no filter is installed, the imagingsystem can be used as a visible light imager to provide full 360 degreeRGB imagery in addition to point clouds.

A laser projection unit consists of a solid-state laser diode, laserpulsing circuitry, aspheric collimation lens, beam splitter, smallrotating mirror, and laser line lens. The laser circuitry pulses thelaser while also providing a frame trigger to each imager. The laserlight is collimated into a beam 403 and 404 using a small aspheric lensdirectly in front of the laser. The laser beam is then split into anupward and downward beam 403 and 404. Each beam 403 and 404 is reflectedoff a small rotating mirror coupled to a laser line lens. The upwardbeam 403 creates a laser line that extends from horizontal to positive80 degrees pitch, while the downward beam 404 creates a laser line thatextends from horizontal to negative 80 degrees pitch.

The proposed field-of-view (shown in FIG. 4) shows the field-of-view ofthe projected lines 403 and 404. FIG. 5 shows the combined field-of-viewof the cameras 405 and 406 and laser projectors 308.

The structured light sensor will be able to measure 360 degreeshorizontally and 160 degrees vertically. At each point in time, thesensor will generate approximately 2080 vertical range measurements.With each imager capturing approximately 180 images/second, the sensorwill be able to generate over 370 k points per second.

The yaw scan rate can be varied, depending upon the current missionneeds. The sensor can be operated with a fine yaw resolution and slowscan rate, providing detailed scans of the environment; or, the sensorcan be operated with a faster yaw rate, providing faster updates at acoarser rate.

Since this device relies on triangulation, the range accuracy will bedependent on range. The expected range error 600 is shown in FIG. 6 ingraph format.

A second approach is to use a time-of-flight line sensor to perform thesame task as shown with the structured light sensor. The line sensorscan be organized as seen in FIGS. 7a and 7 b.

One more possible configuration is the same as shown in FIGS. 7a and 7b, but with the vertical sensing plan 700 aligned with the direction oftravel 701.

The system is composed of a quadrotor, or other UAV, and one or morerange sensors that are used to sense the surrounding environment.

Thus, it is appreciated that the optimum dimensional relationships forthe parts of the invention, to include variation in size, materials,shape, form, function, and manner of operation, assembly and use, aredeemed readily apparent and obvious to one of ordinary skill in the art,and all equivalent relationships to those illustrated in the drawingsand described in the above description are intended to be encompassed bythe present invention.

Furthermore, other areas of art may benefit from this method andadjustments to the design are anticipated. Thus, the scope of theinvention should be determined by the appended claims and their legalequivalents, rather than by the examples given.

The embodiments of the invention in which an exclusive property orprivilege is claimed are defined as follows:
 1. A sensing device forUAVs, comprising: a UAV; a structured light sensor; the structured lightsensor configured to use the size of the quadrotor, in order to providea disparity requirement; and a computer or microprocessor to process thestructured light sensor information; and the computer or microprocessorsending the structured light sensor information to one or morerecipients.
 2. The sensing device for UAVs of claim 1, wherein theprocessing is used for obstacle avoidance.
 3. The sensing device forUAVs of claim 1, wherein the processing is used for mapping thesurroundings.
 4. The sensing device for UAVs of claim 1, wherein the UAVis a quadrotor.
 5. The sensing device for UAVs of claim 1, wherein thestructured light sensor is rotated; and the rotation is accomplished bya mechanism on the vehicle.
 6. The sensing device for UAVs of claim 1,wherein the structured light sensor is rotated; and the rotation isaccomplished by moving the body of the vehicle.
 7. The sensing devicefor UAVs of claim 1, wherein the structured light sensor is rotated; andthe rotation is accomplished by at least one of a mechanism on thevehicle and moving the body of the vehicle, or a combination of the two.8. The sensing device for UAVs of claim 1, wherein multiple lines areused, one horizontal line and one vertical line, to increase thecoverage.
 9. The sensing device for UAVs of claim 1, further comprisinga time-of-flight sensor.
 10. A sensing device for UAVs, comprising aquadrotor; one or more line time-of-flight sensors; a computer ormicroprocessor to process range information; and the computer ormicroprocessor sending the range information to one or more recipients.11. The sensing device for UAVs of claim 10, wherein the processing isused for obstacle avoidance.
 12. The sensing device for UAVs of claim10, wherein the processing is used for mapping the surroundings.
 13. Thesensing device for UAVs of claim 10, wherein the line time-of-flightsensor is rotated; and the rotation is accomplished by a mechanism onthe vehicle.
 14. The sensing device for UAVs of claim 10, wherein theline time-of-flight sensor is rotated; and the rotation is accomplishedby moving the body of the vehicle.
 15. The sensing device for UAVs ofclaim 10, wherein the line time-of-flight sensor is rotated; and therotation is accomplished by at least one of a mechanism on the vehicleand moving the body of the vehicle, or a combination of the two.
 16. Thesensing device for UAVs of claim 10, further comprising a structuredlight sensor.
 17. The sensing device for UAVs of claim 16, whereinmultiple lines are used, one horizontal line and one vertical line, toincrease the coverage.
 18. The sensing device for UAVs of claim 10,wherein the UAV is a quadrotor.
 19. A sensing device for UAVs,comprising: a plurality of fisheye cameras; the cameras are separatedfrom each other to provide parallax; four, 185 degree field-of-viewcameras provide overlapping views over nearly the whole unit sphere; aplurality of laser line scanners; the near-infrared laser projectionunit sends light out into the environment, which is reflected and viewedby the cameras; the laser projection system creates vertical lines,while the cameras will be displaced from each other horizontally’ thisrelative shift (stereo disparity) of the lines, as viewed by differentcameras, enables the lines to be triangulated in 3D space; at each pointin time, a vertical stripe of the world will be triangulated; over time,the laser line will be rotated over all yaw angles to provide full 360degree range sensing capabilities; the two laser line projectors areused to create a line that can then be sensed with the omnidirectionalcameras; each imager is composed of a camera module, a spectral filter,and a wide-angle compound lens; an optical bandpass filter can beinstalled to attenuate incoming ambient light; if no filter isinstalled, the imaging system can be used as a visible light imager toprovide full 360 degree RGB imagery in addition to point clouds; a laserprojection unit consists of a solid-state laser diode, laser pulsingcircuitry, aspheric collimation lens, beam splitter, small rotatingmirror, and laser line lens; the laser circuitry pulses the laser whilealso providing a frame trigger to each imager; the laser light iscollimated into a beam using a small aspheric lens directly in front ofthe laser; the laser beam is then split into an upward and downwardbeam; each beam is reflected off a small rotating mirror coupled to alaser line lens; the upward beam creates a laser line that extends fromhorizontal to positive 80 degrees pitch; the downward beam creates alaser line that extends from horizontal to negative 80 degrees pitch;the structured light sensor will be able to measure 360 degreeshorizontally and 160 degrees vertically; at each point in time, thesensor will generate approximately 2080 vertical range measurements;each imager capturing approximately 180 images/second, the sensor willbe able to generate over 370 k points per second; the yaw scan rate canbe varied, depending upon the current mission needs; the sensor can beoperated with a fine yaw resolution and slow scan rate, providingdetailed scans of the environment; or, the sensor can be operated with afaster yaw rate, providing faster updates at a coarser rate; and sincethis device relies on triangulation, the range accuracy will bedependent on range.
 20. The sensing device for UAVs of claim 19,comprising: a UAV; one or more range sensors that are used to sense thesurrounding environment; a time-of-flight line sensor to perform thesame task as shown with the structured light sensor; and a verticalsensing plan aligned with the direction of travel.